Controlled Warm Air Capture

ABSTRACT

A system for providing air circulation to rack-mounted computers can include a plurality of rack-mounted computers on a plurality of motherboards, each motherboard having a front end near a work space and a back end; one or more fans associated with the plurality of motherboards and adapted to deliver heated air from the computers to a common warm-air plenum; and one or more motor controllers coupled to the plurality of fans and adapted to maintain a substantially constant sensed exhaust temperature for air moved by the fans.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of, and therefore claims priorityunder 35 U.S.C. §120 to, U.S. patent application Ser. No. 11/757,316filed on Jun. 1, 2007, and entitled “Controlled Warm Air Capture,” whichclaims the benefit of prior U.S. Provisional Patent Application Ser. No.60/810,452, filed on Jun. 1, 2006. The entire contents of these priorapplications are incorporated herein by reference.

TECHNICAL FIELD

This document relates to systems and methods for providing cooling forareas containing electronic equipment, such as computer server rooms andserver racks in computer data centers.

BACKGROUND

Computer users often focus on the speed of computer microprocessors(e.g., megahertz and gigahertz). Many forget that this speed often comeswith a cost—higher power consumption. For one or two home PCs, thisextra power may be negligible when compared to the cost of running themany other electrical appliances in a home. But in data centerapplications, where thousands of microprocessors may be operated,electrical power requirements can be very important.

Power consumption is also, in effect, a double whammy. Not only must adata center operator pay for electricity to operate its many computers,but the operator must also pay to cool the computers. That is because,by simple laws of physics, all the power has to go somewhere, and thatsomewhere is, in the end, conversion into heat. A pair ofmicroprocessors mounted on a single motherboard can draw hundreds ofwatts or more of power. Multiply that figure by several thousand (ortens of thousands) to account for the many computers in a large datacenter, and one can readily appreciate the amount of heat that can begenerated. It is much like having a room filled with thousands ofburning floodlights. The effects of power consumed by the critical loadin the data center are often compounded when one incorporates all of theancillary equipment required to support the critical load.

Thus, the cost of removing all of the heat can also be a major cost ofoperating large data centers. That cost typically involves the use ofeven more energy, in the form of electricity and natural gas, to operatechillers, condensers, pumps, fans, cooling towers, and other relatedcomponents. Heat removal can also be important because, althoughmicroprocessors may not be as sensitive to heat as are people, increasesin temperature can cause great increases in microprocessor errors andfailures. In sum, a data center requires a large amount of electricityto power the critical load, and even more electricity to cool the load.

SUMMARY

This document describes systems and methods that may be employed toremove heat efficiently from areas supporting electronic equipment, suchas data centers. In certain implementations, circulation fans may beassociated with particular boards or trays in a rack, and may becontrolled to maintain a near-constant temperature for air exiting eachof the boards. Fans may also be provided in a similar manner for a groupof boards, such as a pair of adjacent boards. The heat from the boards,including from one or more racks that each include a plurality ofboards, may be routed to a warm-air-plenum, through cooling coils, andinto a workspace in a data center where people monitor and attend to thevarious computers. The air may then be circulated back across theboards, such as through open-front cabinets that make up the racks. Theair may then continue to circulate over and over. The air on the backside of each tray may be maintained, by main circulating fans, at thesame pressure (or near the same pressure) as the air on the front sideof each tray, so that the trays can better control their own airflow,particularly at low flow rates.

Providing for control of the exhaust temperature of air leaving theboards, and holding the heated air in a separate space apart from theworkspace can create separate zones having high associated thermalmass—i.e., the warm plenum and related areas, and the cooler workspaceand related areas. As a result, these masses of air act like thermalcapacitors. The warm air in the plenum area can be cooled more readilythan could cooler air because the difference in temperature between thewarm air and cooling water that is used to cool the air will be greaterthan it otherwise would be. In principle, the level of heat transferbetween two zones (e.g., hot air and cooling water) is proportional tothe difference in temperature between the zones, so that increasing thetemperature difference also increases the heat transfer.

By controlling the board (e.g., server) exhaust temperatures, thosetemperatures may be driven upward by slowing the circulation of airacross the board, thereby improving the heat transfer between the warmedair and any cooling fluid even more. Although one may seek to coolelectronic components by forcing as much cool air as fast as possibleover the components, more efficient (and still sufficient) cooling canbe achieved at the system level by going in the opposite direction byslowing the circulation.

Such slow circulation may be aided by operating fans that pull air outof the warm air plenum and drive the air through the cooling coil sothat each tray faces a constant pressure difference across it. As such,the fans on the trays will experience predictable operating parametersand can be operated relatively reliably at very low speeds and volumes.Also, controlling individual units such as a tray or group of trays tomaintain a set exit temperature (and by extension, a set temperaturerise), and controlling more general fans to maintain a set pressuredifference (rather than temperature controlling those fans to atemperature or temperature difference), which may be zero, may helpprevent the two components of the overall system from “fighting” witheach other.

Various implementations may also provide for one or more of thefollowing advantages. Simplified cooling schemes may be realized bymaintaining constant exit temperatures for the various electroniccomponents in a system. Also, elevated temperatures may be achieved inthe air exiting the computer boards or trays, to permit for easier andmore efficient cooling of that air. Moreover, because air may beprovided to a workspace around the electronic equipment at anear-ambient temperature, much less mixing of air having varyingtemperatures occurs in the workspace than may be the case with otherapproaches to data center cooling. Finally, capturing the server exhaustin a warm air plenum and treating (cooling) it before releasing it intoa facility results in much more predictable and uniform air temperaturesat the inlet of the servers. This allows one to design for a loweraverage inlet temperature, rather than the peaks associated withreleasing uncooled air from the servers back into the facility. Inaddition, segregation of warm air zones from cold air zones may helpprevent thermal short circuits, where localized warm air may be pulledback through a rack.

In addition, because the air in the warm-air plenum and the workspacemay serve as thermal capacitors that absorb changes in the system,changes in the conditions of the system, such as failed components,become less important. Also, other components of the system may bearranged so as to mask changes in the system, such as changes caused bythe failure of certain components in the system. For example, if aparticular cooling coil or fan fails or needs to be changed, theremaining coils or fans in a data room may make up the difference. Inaddition, use of common arrangements and control schemes across manycomponents, such as for all boards in a data center, may allow equipmentto be acquired as stock, rather than specialized, equipment, therebylowering acquisition costs, and permitting more modular implementationsthat allow for efficient use of repetitive components. In addition,individual fans associated with particular trays in a rack of computersmay be eliminated, with system fans providing motive force. Also,maintaining a relatively warm workspace (over or well over 75° and evenwarmer) or cold aisle, and an even warmer warm aisle can provide, incertain implementations (as described below), the advantages of nothaving to perform latent heat removal, and allowing different coolingplant technologies to dominate (cooling towers instead of chillers). Inaddition, elevated warm aisle, or warm air plenum, temperature canreduce the size of a cooling plant.

In some implementations, a system for providing air circulation torack-mounted computers includes a plurality of rack-mounted computers ona plurality of motherboards, each motherboard having a front end near awork space and a back end; one or more fans associated with theplurality of motherboards and adapted to deliver heated air from thecomputers to a common warm-air plenum; and one or more motor controllerscoupled to the plurality of fans and adapted to maintain a substantiallyconstant sensed exhaust temperature for air moved by the fans.

Each of the plurality of fans can be mounted to a power supply to drawheated air through the power supply. The system can further includewalls extending from a top surface of each of the plurality ofmotherboards and arranged to direct air passing over each of theplurality of motherboards to the plurality of fans. The walls canseparate low heat-generating computer components from highheat-generating components that are located in a flow path for each ofthe fans. One or more of the fans can deliver air from more than onemotherboard. The system can further include a plurality of open-frontcabinets to hold the plurality of motherboards.

The system can further include a warm-air plenum, wherein the warm-airplenum is configured to receive warm exhaust air from a plurality ofracks of computers. The system can further include a fan-coilarrangement in fluid communication with the warm-air plenum to cool airfrom the warm-air plenum and to deliver the cooled air to the workspace. The warm air plenum can include an attic space above the workspace.

The system can further include one or more pressure sensors connected toone or more motor controllers and configured to maintain a set pressuredifference between an air intake for the motherboards and an air exitfor the motherboards. The set pressure difference can be substantiallyzero.

In some implementations, the sensed exhaust temperature exceeds 110degrees Fahrenheit. The system can further include a system controlledconfigured to maintain the temperature in the work space below 78degrees Fahrenheit, and the exhaust temperature in excess of 110 degreesFahrenheit.

In some implementations, a cooling method includes providing cooling airto a human operator workspace adjacent to a plurality of computers;drawing air from the workspace across the plurality of computers andinto a common warm-air plenum; and controlling a plurality of fansdrawing the air to maintain a substantially constant temperature of theair passing through the fans at a constant temperature.

Each fan of the plurality of fans can be associated with one computermotherboard. Each of the plurality of fans can be set to maintain acommon temperature. The cooling air can be provided to the workspace bya common fan-coil unit. The method can further include controlling thefan-coil unit to maintain a set temperature in the workspace. In someimplementations, the fans are controlled to maintain a temperature of104-122 degrees Fahrenheit for air entering the common warm-air plenum,and the fan-coil unit is controlled to maintain a temperature of 70-86degrees Fahrenheit for air entering the workspace. The air can be drawnfrom the workspace through the front of one or more open-front cabinetsthat face the workspace and that house the computers.

In some implementations, a system for removing heat from a volumesupporting electronic equipment includes one or more enclosures definingan interior space and an exterior workspace; a plurality of supports forholding heat-generating electronic devices in the interior space; an airmoving device positioned to create a flow of air in the enclosure acrossthe electronic devices and to maintain the air temperature through theair moving device at a set temperature; and one or more heat exchangersin the interior space located to receive substantially only air heatedby the electronic devices, to cool the air, and to return the cooled airfor recirculation across the electronic devices.

The system can further include one or more air distributors to pass thecooled air to the exterior workspace. The system can further include anair intake opening in the enclosure, wherein the plurality of heatgenerating devices are positioned downstream from the opening. Thesystem can further include a plurality of heat generating electronicdevices attached to the supports and mounted on a plurality of spaced,substantially planar boards, and the flow of air occurs in the spacesbetween adjacent pairs of boards. The air moving device and heatexchanger can be configured to maintain the air above the air's dewpoint under expected standard operating conditions. The air movingdevice and heat exchanger can be configured to maintain the exteriorworkspace temperature below 100 degrees Fahrenheit when the interiorspace temperature is substantially above 120 degrees Fahrenheit

In some implementations, the expected operating conditions include aheated air temperature of approximately 45 degrees Celsius and a cooledair temperature of approximately 25 degrees Celsius. The expectedoperating conditions can further include a supply water temperature tothe heat exchanger of approximately 20 degrees Celsius and a returnwater temperature of approximately 40 degrees Celsius.

The plurality of supports can be part of a vertical rack server cabinet,and the one or more enclosures comprise a plenum attached to a side ofthe cabinet. The air moving device can include a fan located downstreamof the heat exchanger.

In some implementations, a system for removing heat from a volumesupporting a plurality of computers includes a rack server cabinetconfigured to hold a plurality of computer motherboards that arepositioned to permit airflow over components mounted on themotherboards; a plenum having an air intake side attached to a side ofthe rack server cabinet, the plenum configured to capture airflow overthe components; a fan between the components and the plenum having acontroller set to maintain a constant temperature for air entering theplenum; and a heat exchanger in fluid communication with an air outputside of the plenum to receive and cool heated air that has passed overthe components and recirculate the cooled air back to the cabinet.

In some implementations, a method for removing heat from a volumesupporting electronic equipment can include creating an airflow from anambient space over a plurality of computers and controlling a rate ofairflow to maintain a temperature of air exiting the plurality ofcomputers; capturing heated air from the airflow in a plenum; coolingthe captured heated air; and returning the cooled air to the ambientspace

The step of cooling the heated air can include cooling the air from atemperature above 45 degrees Celsius to a temperature below 25 degreesCelsius. The step of creating an airflow from an ambient space over aplurality of computers can include drawing air from a computer roomacross an opening in a side of a rack server. The method can furtherinclude capturing airflow from a plurality of racks of computers in acommon plenum. The method can further include mixing the heated air inthe plenum with outside air when the outside air is lower in temperaturethan the cooled air

In some implementations, a system for removing heat from a volumesupporting electronic equipment includes one or more enclosures definingan interior space and an exterior space; a plurality of heat generatingelectronic devices in the interior space; and means for cooling the airheated by the electronic devices and for re-circulating the air acrossthe electronic devices

In some implementations, a method for cooling electronic equipmentincludes drawing air into a computer enclosure from a human operatorworkspace adjacent to a plurality of computers in the computerenclosure; and controlling the flow rate of the air across a computer toobtain a temperature rise across the computer of greater than 18 degreesCelsius.

The method can further include controlling the flow rate of the airacross the computer so that the air temperature is below but close to anexpected failure temperature of a component in the computer having alowest failure temperature of the components needed to operate thecompute.

In some implementations, a method for cooling electronic equipmentincludes drawing air into a computer enclosure from a human operatorworkspace adjacent to a plurality of computers in the computerenclosure; and controlling the flow rate of the air across a computer tomaintain a temperature so that the air temperature is below but close toan expected failure temperature of a component in the computer having alowest failure temperature of the components needed to operate thecomputer.

In some implementations, a data center system includes a shippingcontaining defining an interior volume; two rows of a plurality ofcomputer server racks having front ends on opposed sides of a centralaisle in the volume, each of the server racks open to the central aisleand having a top, bottom, and two sides defining a rack perimeter; andair circulation fans adjacent the rack perimeters of the server racksand configured to release cooled air to the front of the racks.

The system can further include warm air plenums at back ends of thecomputer server racks. The warm air plenums can each be in communicationwith a plurality of racks. The system can further include a plurality ofcooling coils defining a boundary of the warm air plenum. The aircirculation fans can pull air through the cooling coils and direct theair into the central aisle. The air circulation fans can release the airnear planes defined by the front ends of the computer server racks. Thesystem can further include a plurality of computers mounted in theplurality of server racks. The system can further include power and dataconnections on front side of the plurality of computers adjacent thecentral aisle.

The details of one or more embodiments are set forth in the accompanyingdrawings and the description below. Other features, objects, andadvantages will be apparent from the description and drawings, and fromthe claims.

DESCRIPTION OF DRAWINGS

FIG. 1 a shows a plan view of a tray in a rack-mount computer system.

FIG. 1 b shows a front view of the tray in FIG. 1 a.

FIG. 1 c shows a side view of the tray in FIG. 1 a.

FIG. 1 d shows a plan view of a tray in a rack-mount computer system,having dual-zone power supply ventilation.

FIG. 1 e shows a front view of the tray in FIG. 1 d.

FIG. 1 f shows a side view of the tray in FIG. 1 d.

FIG. 1 g shows a plan view of a tray in a rack-mount computer system,having dual-zone adjustable power supply ventilation.

FIG. 2 a shows a plan view of a data center in a shipping container.

FIG. 2 b shows a sectional view of the data center from FIG. 2 a.

FIG. 2 c shows a perspective view of a modular computing environment forhousing a data center or part of a data center.

FIG. 3 a shows a plan view of a data center.

FIG. 3 b shows a sectional view of the data center from FIG. 3 a.

FIG. 4 a shows a plan view of a data center.

FIG. 4 b shows a sectional view of the data center from FIG. 4 a.

FIG. 5 a shows a plan view of a data center.

FIG. 5 b shows a sectional view of the data center from FIG. 5 a.

FIG. 5 c shows a sectional view of another implementation of a datacenter.

FIG. 6 is a flowchart showing actions for an exemplary operation ofcooling components in a data center.

FIG. 7 shows plan views of two exemplary trays for use in a rack-mountcomputer system.

Like reference symbols in the various drawings indicate like elements.

DETAILED DESCRIPTION

FIG. 1 a shows a plan view of a tray 10 in a rack-mount computer system,while FIG. 1 b shows a front view, and FIG. 1 c shows a side view, ofthe tray 10 in FIG. 1 a. The term “tray” is not limited to anyparticular arrangement, but instead includes any arrangement ofcomputer-related components coupled together to serve a particularpurpose, such as on a motherboard. Trays may be generally mountedparallel to other trays in a horizontal or vertical stack, so as topermit denser packing than would otherwise be possible with computershaving free-standing housings and other components. The term “blade” mayalso be employed to refer to such apparatuses, though that term tooshould not be limited to a particular arrangement. Trays may beimplemented in particular configurations, including as computer servers,switches (e.g., electrical and optical), routers, drives or groups ofdrives, and other computing-related devices. In general, the trays in asystem take a standardized physical and electrical form to be easilyinterchangeable from one location in the system to another, but thetrays may take other appropriate forms.

In general, the tray 10 may include a standard circuit board 12 on whicha variety of components are mounted. The board 12 may be arranged sothat air enters at its front edge (to the left in the figure), is routedover a number of heat generating components on the board 12, and isdrawn through a power supply 14 and fan 16 before being exhausted fromthe tray 10. The fan 16 may also be arranged to push air through thepower supply 14. In addition, the fan 16 may be located in otherpositions relative at other positions along the back edge of the trayand at locations away from a back edge of the tray 10. The power supply14 may likewise be positioned at other locations and need not be joinedto the fan 16.

In this arrangement, the heat from power supply 14 may be picked upafter the heat from other components on the board 12 is picked up by theair flow. In this manner, the speed of fan 16 may be controlled tomaintain a set temperature for the air exiting the board 12, or fortemperatures at other points on the tray 10. For example, a thermocoupleor other sort of temperature sensor may be placed in the air flow, suchas upstream of the power supply 14 or downstream of the fan 16, and thefan speed may be modulated to maintain a set temperature. Thetemperature of the exiting air may also be highly elevated compared tosystems that do not control airflow in this manner. It may be moreefficient to cool this air than it would be to cool air that does nothave such an elevated temperature.

Air may be routed over board 12 by walls 26 a, 26 b, 26 c. Wall 26 a mayblock one side of board 12, and may funnel air toward openings in powersupply 14. Where the walls 26 a, 26 c do not taper, the air mayotherwise be directed to the fan 16. Wall 26 c may block one side ofboard 12, so as to prevent air from moving directly from the workspaceinto an area behind tray 10 (i.e., to the right in the figure). Forexample, a plenum may be provided behind multiple boards in the form ofan open wall into which the boards may be placed, or in the form of awall having multiple openings into which fans may be slid. In certainimplementations, fully blocking or sealing of such a plenum may not benecessary, such as when the pressure difference between the plenum andthe workspace is minimal.

Wall 26 b separates one portion of tray 10 from another. In particular,wall 26 b separates the portion of tray 10 containing heat generatingcomponents, such as microprocessors 21 a, 21 b, from components thatgenerate substantially less heat, such as hard drives 18 a, 18 b. Inmaking such a separation, wall 26 b substantially blocks airflow overthe components that generate less heat, and increases airflow over theheat generating components. In addition, wall 26 b is arranged to routeairflow into openings in power supply 14. Although not pictured, wall 26b may block areas on tray 10 but may provide that each blocked area(i.e., the area on each side of wall 26 b) may still be in fluidcommunication with fan 16. For example, fan 16 may be designed to haveopenings that will lie on each side of wall 26 b, and the openings maybe sized or otherwise tuned so as to provide for relative levels of airflow on the opposing sides of the wall 26 b. The “tuning” of the airflow may be made to match the relative thermal load of components oneach side of wall 26 b, so that more air flows on the side of wall 26 bhaving the most thermal load, or that otherwise requires more cooling.

Board 12 may hold a variety of components needed in a computer system.As shown, board 12 holds a dual processor computer system that usesprocessor 21 a and processor 21 b connected to a bank of memory 24. Thememory 24 may be in the form, for example, of a number of single in-linememory modules (SIMMs), dual in-line memory module (DIMMs), or otherappropriate form. Other components of the computer system, such as chipsets and other chips, have been omitted for clarity in the figure, andmay be selected and arranged in any appropriate manner.

Board 12 may also be provided with connections to other devices. Networkjack 22, such as in the form of an RJ-45 jack or an optical networkingconnection, may provide a network connection for tray 10. Otherconnections may also be provided, such as other optical networkingconnections, video output connections, and input connections such askeyboard or pointing device connections (not shown).

Impingement fans 20 a, 20 b may be mounted above each microprocessor 21a, 21 b, to blow air downward on the microprocessors 21 a, 21 b. In thismanner, impingement fans 20 a, 20 b may reduce boundary layer effectsthat may otherwise create additional heat buildup on microprocessors 21a, 21 b. As a result, lateral airflow across tray 10 can be reduced evenfurther, while still adequately controlling the temperature rise to themicroprocessors 21 a, 21 b.

Other heat relief mechanisms may also, or alternatively, be provided formicroprocessors 21 a, 21 b. For example, one or more heat sinks may beprovided, such as in the form of certain finned, thermally conductivestructures. The heat sinks may be directly connected to microprocessors21 a, 21 b, or may be located to the sides of microprocessors 21 a, 21b, and may be attached by heat pipes to plates mounted to the top ofmicroprocessors 21 a, 21 b. Thermally conductive grease or paste may beprovided between the tops of microprocessors 21 a, 21 b, and any heatsinks to improve heat flow out of microprocessors 21 a, 21 b.

In operation, tray 10 may be mounted flat horizontally in a server racksuch as by sliding tray 10 into the rack from the rack front, and over apair of rails in the rack on opposed sides of the tray 10—much likesliding a lunch tray into a cafeteria rack, or a tray into a bread rack.Tray 10 may alternatively be mounted vertically, such as in a bank oftrays mounted at one level in a rack. The front of the rack may be keptopen to permit easy access to, and replacement of, trays and to permitfor air to flow over the tray 10 from a workspace where technicians orother professionals operating a data center may be located. In thiscontext, the term workspace is intended to refer to areas in whichtechnicians or others may normally be located to work on computers in adata center.

After sliding a tray 10 into a rack, a technician may connect a tray toappropriate services, such as a power supply connection, batteryback-up, and a network connection. The tray 10 may then be activated, orbooted up, and may be communicated with by other components in thesystem.

Although tray 10 is shown in the figures to include a multi-processorcomputer system, other arrangements may be appropriate for other trays.For example, tray 10 may include only hard drives and associatedcircuitry if the purpose of the tray is for storage. Also, tray 10 maybe provided with expansion cards such as by use of a riser modulemounted transversely to the board 12. Although particular forms of tray10 may be provided, certain advantages may be achieved in appropriatecircumstances by the use of common trays across a rack or multipleracks. In particular, great efficiencies may be gained by standardizingon one or a small handful of trays so as to make interaction betweentrays more predictable, and to lower the need to track and store manydifferent kinds of trays.

A data center may be made up of numerous trays (hundreds or thousands),each mounted in one of numerous racks. For example, several dozen traysmay be mounted in a single rack within a space, with approximatelyseveral inches between each tray. As explained in more detail below,each of the trays in a rack may back up to a warm air plenum thatreceives exhaust air from the trays and routes that air to a coolingunit that may re-circulate the air into the workspace in front of theracks.

Trays may also be packaged in groups. For example, two stacked trays maybe matched as a pair, with one fan 16 serving both trays (not shown).Specifically, the fan 16 may be approximately double the height anddiameter of a single tray unit, and may extend from the lower tray in apair up to the top of the upper tray in a pair. By such an arrangement,the slowest turning portions of the fan, in the fan center, will be nearthe board of the top tray, where less airflow will normally occurbecause of boundary layer effects. The larger and faster moving portionsof the fan 11 will be located nearer to the free areas of each tray 10so as to more efficiently move air over the trays and through therespective power supplies more freely. In addition, a double-height fanmay be able to move more air than can a single-height fan, at lowerrotation speeds. As a result, a fan in such an arrangement may produceless noise, or noise at a more tolerable frequency, than could a smallerfan. Parallel fans may also be used to increase flow, and serial fansmay be used to increase pressure, where appropriate.

Fan 16 may be controlled to maintain a constant temperature for airexiting fan 16 or at another point. By locating fan 16 downstream ofpower supply 14, and power supply 14 downstream of the other componentsof tray 10, the arrangement may maximize the heat rise across tray 10,while still maintaining adequately low temperatures for heat-sensitivecomponents mounted to board 12, such as microprocessors 21 a, 21 b.Also, the power supply 14 may be less sensitive to higher temperaturesthan are other components, and so may be best located at the end of theair flow, where the temperatures are highest.

Although many applications seek to substantially increase airflow acrossheat generating components so as to increase the rate of heatdissipation from the components, the arrangement pictured here allowsairflow across tray 10 to be slowed substantially to increase thetemperature rise across tray 10. Increasing the temperature risedecreases the mass flow rate, and can make cooling across the entiresystem more efficient.

In particular, when the temperature of the warm exiting air isincreased, the difference in temperature between the warm air and anycooling water entering a cooling coil to cool the warm air, alsoincreases. The ease of heat transfer is generally directly proportionalto this difference in temperature. Also, when the difference intemperature is relatively small, increasing the difference by only oneor two degrees can produce a substantial increase in the amount of heatexchange between the warm air and the cooling water. As a result, asystem run at higher exhaust temperatures from board 12 can providesubstantial advantages in efficiency, and lower energy consumption.

In certain embodiments, the temperature rise across tray 10 may beapproximately 20° C. As one example, air may enter the space above board12 from a workspace at 25° C., and may exit fan 16 at 45° C. Theentering temperature may also be about 21-30° C. (70-86° F.), and theexiting temperature 40-50° C. (104-122° F.). The 45° C. exhausttemperature or other temperature may be selected as a maximumtemperature for which the components in tray 10 can be maintainedwithout significant errors or breakdowns, or a safe temperature ofoperation. The 25° C. entering temperature or other temperature may be atemperature determined to create a comfortable or tolerable temperaturein the workspace in a data center. The entering temperature may also belinked to a maximum allowable temperature, such as a federal or stateOSHA-mandated maximum temperature. The entering temperature could beapproximately 40° Celsius, which matches certain limits established bybodies governing workplace safety.

In other implementations, air may enter the space above board 12 at atemperature of 50° C., where appropriate thermal removal mechanisms ormethods are provided for the components on board 12. For example,conductive and liquid-cooled components may be placed in contact withmicroprocessors 21 a, 21 b to increase the rate of heat dissipation fromthose components. Where a higher input temperature is selected, thetemperature difference across tray 10 will generally be lower than if alower input temperature is selected. However, heat will be easier toremove from such heated air when it passes through a cooling coil.Higher temperatures for expected breakdowns include components thattolerate case temperatures of 85 degrees Celsius. In addition, the exitair temperature from tray 10 may be as high as 75 degrees Celsius. Anoutput temperature may be most easily controlled by locating atemperature sensor at the actual output (or aiming it at the actualoutput). Such an output temperature may also be controlled or maintainedwithin an acceptable temperature range by placing a temperature sensorat a location away from the output, but where the difference intemperature is adequately predictable.

In the front view of FIG. 1 b, one can see power supply 14 located atthe back of tray 10, and perforated to permit the flow of air throughpower supply 14. In addition, one can see hard drive 18 a located in anarea walled off from the heat generating components of tray 10 by wall26 b. As noted above, the power supply 14 could also be situated so asto receive air leaving two different zones on tray 10, with the powersupply 14 or other components tuned to maintain certain relative airflow rates from each side.

The side view of FIG. 1 c shows more clearly the relationship of theimpingement fans 20 a, 20 b and microprocessors 21 a, 21 b. The fans 20a, 20 b are shown schematically for clarity. Air is pulled through thetops of fans 20 a, 20 b, and driven down against the top ofmicroprocessors 21 a, 21 b. This process breaks up layers of warm airthat may otherwise form above microprocessors 21 a, 21 b.

As noted above, other techniques for spot removal of heat fromcomponents such as microprocessors 21 a, 21 b may also be employed. Asone example, heat sinks may be attached on top of or to the side ofmicroprocessors 21 a, 21 b, and may be cooled by circulating air or aliquid, such as water or fluorinert liquid, or oils. Liquid supply andreturn tubes may be provided down each rack, with taps at which toconnect pipes for cooling particular components. Circulation of liquidto the components may be driven by pressure created centrally in thesystem (e.g., from natural tap water pressure or large pumps) or bysmall pumps local to a particular tray 10. For example, smallperistaltic, centrifugal, vane or gear-rotor pumps may be provided witheach tray to create liquid circulation for the tray 10.

Alternatively, a portion of a rack or a component associated with a rackmay be cooled, such as by passing liquid through passages in thecomponent. Heat sinks for each heat generating component may then becoupled physically to the cooled component in the rack so as to drawheat out of the components on the tray 10 and into the rack. As oneexample, a vertical runner on the rack may be provided with clamps intowhich heat pipes attached to heat-generating components on tray 12 arereceived, so that the heat pipes may pull heat away from thosecomponents and into the runner. The runner may further include fluidpassages to carry cooling fluid. Thus, the runner will be kept cool, andwill draw heat by conduction from the heat-generating components.

FIG. 1 d shows a plan view of a tray in a rack-mount computer system,having dual-zone power supply ventilation. FIG. 1 e shows a front viewof the tray in FIG. 1 d. FIG. 1 f shows a side view of the tray in FIG.1 d. The general arrangement of components on the tray 12 here issimilar to that in FIGS. 1 a-1 c, although the particular arrangementand layout of components is not generally critical. However, in thesefigures, the wall 26 b has its rear edge pulled forward from the backwall of the tray 12. Also, the power supply 14 has two areas ofopenings—one on its front edge, as can be seen in FIG. 1E, and one onits side edge, as can be seen in FIG. 1F. The openings on the front edgegenerally provide ventilation for the hot side of the tray 12, whilethose on the side edge provide ventilations for the cool side of thetray 12.

The openings may be sized or otherwise organized to provide particularapproximate levels of ventilation to each side of the tray 12. As can beseen in FIGS. 1E and 1F, the front edge of the power supply 1 has moreholes than does the edge; in addition, the air flow from the front edgeis straight, while air coming in through the side edge needs to curve.As a result, the front edge will provide a higher level of ventilationthan will the side edge, and will thus be able to carry away the higherlevel of heat generated on the hot side of tray 12. The amount of aircarried on a hot side might also be lower than on a cool side, such aswhere equipment requirements force the cool side to stay at a lowtemperature. In other words, in setting flow rates for each portion oftray 12, both heat generation and desired operating temperature may betaken into account.

FIG. 1 g shows a plan view of a tray in a rack-mount computer system,having dual-zone adjustable power supply ventilation. Here, the wall 26b is positioned to direct a certain amount of ventilating air from eachside of wall 26 b. The wall 26 b may be positioned on tray 12 at anappropriate position, and its terminal end may be made adjustablethrough pivoting or other mechanisms, so as to permit on-site adjustmentof air flow.

In addition, gate 27 may be provided over a front surface of powersupply 14 to provide adjustment to the size of openings on the frontsurface via openings in the gate 27 that form an interference patternwith openings on power supply 27 (much like the openings on certainspice containers). The interference pattern may be different for eachside of tray 12, so that moving the gate 27 causes a greater effect onthe airflow for one side of tray 12 than its does for the other side oftray 12.

Temperature-dependent mechanisms may also be provided to control theflow of air through power supply 14. For example, polymer or metallicmaterials that change shape with temperature may be used to formopenings that close as their temperature falls—thereby driving back upthe exit temperature of air from a particular portion of tray 12. As oneexample, the materials may produce a form of stoma that opens andcloses. Also, mechanisms such as temperature-controlled louvers, or atemperature-controlled actuator on gate 27 may be used to controlairflow over board 12. Such air control mechanisms may also be locatedoff of tray 12. For example, a wall perforated by temperature dependentstoma may be placed behind a number of racks filled with trays, and maythereby control the exit temperature for all of the racks in aconvenient manner. In such a situation, as in others discussed herein,fan 16 may be eliminated from tray 12, and a central ventilation systemmay pull air through the various trays and racks.

FIG. 2 a shows a plan view of a data center 200 in a shipping container202. Although not shown to scale in the figure, the shipping container202 may be approximately 40 feet along, 8 feet wide, and 9.5 feet tall.Packaging of a data center in a shipping container may permit for moreflexible and automated data center manufacture, such as by having acentrally-trained crew construct a large number of such data centers. Inaddition, the portability offered by a shipping container permits forquicker and more flexible deployment of data center resources, and thusallows for extension and projection of a network more easily to variousareas.

The container 202 includes vestibules 204, 206 at each end. One or morepatch panels or other networking components to permit for the operationof data center 200 may also be located in vestibules 204, 206. Inaddition, vestibules 204, 206 may contain connections and controls forthe shipping container. For example, cooling pipes (e.g., from heatexchangers that provide cooling water that has been cooled by condenserwater supplied from a source of free cooling such as a cooling tower)may pass through the end walls of a container, and may be provided withshut-off valves in the vestibules 204, 206 to permit for simplifiedconnection of the data center to, for example, cooling water piping.Also, switching equipment may be located in the vestibules 204, 206 tocontrol equipment in the container 202.

A central workspace 208 may be defined down the middle of shippingcontainer 202 as an aisle in which engineers, technicians, and otherworkers may move when maintaining and monitoring the data center 200.For example, workspace 208 may provide room in which workers may removetrays from racks and replace them with new trays. In general, workspace208 is sized to permit for free movement by workers and to permitmanipulation of the various components in data center 200, including toprovide space to slide trays out of their racks comfortably.

A number of racks such as rack 219 may be arrayed on each side ofworkspace 208. Each rack may hold several dozen trays, like tray 220, onwhich are mounted various computer components. The trays may simply beheld into position on ledges in each rack, and may be stacked one overthe other. Individual trays may be removed from a rack, or an entirerack may be moved into workspace 208.

The racks may be arranged into a number of bays such as bay 218. In thefigure, each bay includes six racks and may be approximately 8 feetwide. The data center 200 includes four bays on each side of workspace208. Space may be provided between adjacent bays to provide accessbetween the bays, and to provide space for mounting controls or othercomponents associated with each bay. Various other arrangements forracks and bays may also be employed as appropriate.

Warm air plenums 210, 212 are located behind the racks and along theexterior walls of the shipping container 202. The warm air plenumsreceive air that has been pulled over trays, such as tray 220, fromworkspace 208. The air movement may be created by fans such as fan 16 inFIGS. 1 a-1 c. Where each of the fans on the associated trays iscontrolled to exhaust air at one temperature, such as 45° C., the air inplenums 210, 212 will generally be a single temperature or almost asingle temperature. As a result, there will be little need for blendingor mixing of air in warm air plenums 210, 212.

FIG. 2 b shows a sectional view of the data center from FIG. 2 a. Thisfigure more clearly shows the relationship and airflow between workspace208 and warm air plenums 210, 212. In particular, air is drawn acrosstrays, such as tray 220, by fans at the back of the trays. Althoughshown earlier as fans associated with single trays or a small number oftrays, other arrangements of fans may also be provided. For example,larger fans or blowers, such as air induction blowers, may be providedto serve more than one tray.

Air is drawn out of warm air plenums 210, 212 by fans 222, 224,respectively. Fans 222, 224 may take various forms. In one exemplaryembodiment, fans 222, 224 may be in the form of a number of squirrelcage fans. The fans 222, 224 may be located along the length ofcontainer 202, and below the racks, as shown in the figure. A number offans may be associated with each fan motor, so that groups of fans maybe swapped out if there is a failure of a motor or fan.

An elevated floor 230 may be provided at or near the bottom of theracks, on which workers in workspace 208 may stand. The elevated floor230 may be formed of a perforated material, of a grating, or of meshmaterial that permits air from fans 222, 224 to flow into workspace 208.Various forms of industrial flooring and platform materials may be usedto produce a suitable floor that has low pressure losses.

Fans 222, 224 may blow heated air from warm air plenums 210, 212 throughcooling coils 226, 228. Cooling coils 226, 228 may be sized using wellknown techniques, and may be standard coils in the form of air-to-waterheat exchangers providing a low air pressure drop, such as a 0.1 inchpressure drop. Cooling water may be provided to coils 226, 228 at atemperature, for example, of 20 degrees Celsius, and may be returnedfrom coils 226, 228 at a temperature of 40 degrees Celsius. In otherimplementations, cooling water may be supplied at 15 degrees Celsius or10 degrees Celsius, and may be returned at temperatures of about 25degrees Celsius, 30 degrees Celsius, 35 degrees Celsius, 45 degreesCelsius, 50 degrees Celsius, or higher temperatures. The position of thefans 222, 224 and the coils 226, 228 may also be reversed, so as to giveeasier access to the fans for maintenance and replacement. In such anarrangement, the fans 222, 224 will draw air through the coils 226, 228.

The particular supply and return temperatures may be selected as aparameter or boundary condition for the system, or may be a variablethat depends on other parameters of the system. Likewise, the supply orreturn temperature may be monitored and used as a control input for thesystem, or may be left to range freely as a dependent variable of otherparameters in the system. For example, the temperature in workspace 208may be set, as may the temperature of air entering plenums 210, 212. Theflow rate of cooling water and/or the temperature of the cooling watermay then vary based on the amount of cooling needed to maintain thoseset temperatures.

The particular positioning of components in shipping container 202 maybe altered to meet particular needs. For example, the location of fans222, 224 and coils 226, 228 may be changed to provide for fewer changesin the direction of airflow or to grant easier access for maintenance,such as to clean or replace coils or fan motors. Appropriate techniquesmay also be used to lessen the noise created in workspace 208 by fans222, 224. For example, placing coils in front of the fans may help todeaden noise created by the fans. Also, selection of materials and thelayout of components may be made to lessen pressure drop so as to permitfor quieter operation of fans, including by permitting lower rotationalspeeds of the fans.

Also, the fans may pull air or push air through the coils. Pushing airmay have the advantage of being quieter, as the coils may block out acertain amount of the fan noise. Also pushing of air may be moreefficient. Pulling of air may provide a benefit of allowing a limitednumber of fans to operate on a much larger bank of coils, as all thepulling fans can be connected to a plenum, and may create a relativevacuum behind the coils to pull air through. In such an arrangement, ifone of the fans breaks down, the others can more easily provide supportacross the entire coil length.

Airflow in warm air plenums 210, 212 may be controlled via pressuresensors 209, 211. For example, the fans 222, 224 may be controlled sothat the pressure in warm air plenums 210, 212 is roughly equal to thepressure in workspace 208. The volume on one side of pressure sensors209, 211 may be the warm-air plenum, and the volume on the other sidemay be the workspace 208; where a common barrier between these spaces isnot available, taps may be provided to permit sensing of pressure ineach volume. More precisely, the pressure where the air leaves the tray220 may be kept roughly equal to the pressure where it enters the tray,or a set difference in pressures other than zero may be maintained.Where such a pressure relationship is maintained, each of the fans thatdraw air across the trays 220 (where each tray may have one or morededicated fans) may “see” a uniform and predictable world around it. Asa result, the airflow across each tray 220 may likewise be more uniformand predictable, so that adequate airflow can be maintained, adequatecooling can be maintained by extension, fewer hotspots will arise, andfewer equipment failures will result. In addition, such a system maybetter isolate problems in one area of the system from operations inother areas, so that the system has better “diversity” of operation.

Pressure sensors 209, 211 may be, for example, differential pressuresensors such as the Setra Model 263 differential pressure transducer.Taps for the pressure sensors 209, 211 may be placed in any appropriatelocation for approximating a pressure differential across the trays 220.For example, one tap may be placed in a central portion of plenum 212,while another may be placed on the workspace 208 side of a wallseparating plenum 212 from workspace 208. The pressure differentialbetween workspace 208 and plenums 210, 212 may be maintained, forexample, at about 20/1000ths of an inch of static pressure, with theslight vacuum on the plenum side. In general, the sensors 209, 221 maybe operated in a conventional manner with a control system to controlthe operation of fans 222, 224. For example, variable speed drives maybe used to increase or decrease the speed of the fans 222, 224 tomaintain a particular pressure differential, such as zero or nearly zerodifferential. One sensor may be provided in each plenum, and the fansfor a plenum or a portion of a plenum may be ganged on a single controlpoint.

Controlling pressure in the warm air plenum relative to the pressure onthe intake side of the racks, including when the racks are open to aworkspace, can, in certain configurations, provide one or moreadvantages. For example, each tray can better control its ownenvironment because it operates, airflow-wise, as if it is located in anopen room—it sees a consistent and predictable pressure difference, ofapproximately zero in certain implementations, on each of its ends. As aresult, each tray can more closely match its operation to its own heatloads. In addition, the operation of each tray is also made much lessdependent on the operation of other trays. Multiple groups of trays mayalso be controlled together, if a particular configuration calls forsuch grouping.

By making the entry and exit pressures for a tray or board predictable,fans for the tray or board may be run more slowly, and highertemperature rises (which can result in more efficient system-sidecooling) across the tray or board may be achieved. In contrast, if theplenum fans 222, 224 are controlled based on plenum temperatures, thosefans may be run too fast. For instance, an operator of such a system mayseek to prevent stagnation of airflow on any board so as to preventcomponent burnout. To do so, the operator may seek to ensure that theplenum does not provide backpressure to any board. And to do that, theoperator may set the temperature of the warm air plenum fairly low sothat more air than necessary is pulled by the plenum fans 222, 224. Suchoperation will tend to draw air out of the trays and racks, and maydefeat attempts by individual racks to maintain low air flow rates andhigh temperature rises across the trays or boards.

In addition, pressure is generally consistent across a space, whereastemperature differences may be much more localized. As a result, thelocation of pressure sensors 209, 211, may be less important than wouldbe the particular location of temperature sensors. That may beespecially true at the intake side of the trays, such as in theworkspace 208, because air velocities are relatively slow there and arelikely to create higher temperature differences across space due to lessblending in the slower moving air.

In addition, use of pressure as a control signal for the plenum fans maypermit for lower capital costs. As one example, the warm air plenum maygenerally be made smaller when a system is well balanced, which can saveon floor space and permit more equipment to be installed in a datacenter. Also, with the ability to cover problems with one fan usingother fans, the various fans can be sized smaller and individually lessreliable, and thus acquired for less money. In addition, where the warmair plenum is roughly matched in pressure with the surrounding spaces,such as a workspace, the sealing of components for air leakage is lesscritical, so that savings may be achieved in labor, maintenance, andgoods for such sealing.

Such an implementation also permits for more flexibility in planning andoperating a data center. For planning, various unmatched trays may beinserted into a system without as much concern for how each will respondto the system, or will affect each other, because each simply needs tobe able to control its own individual airflow and deal with what appearsto be a neutral surrounding.

For operation, the system may better isolate problems in one area fromother components. For instance, if a particular rack has trays that areoutputting very warm air, such action will not affect a pressure sensorin the plenum (even if the fans on the rack are running at high speed)because pressure differences quickly dissipate, and the air will bedrawn out of the plenum with other cooler air. The air of varyingtemperature will ultimately be mixed adequately in the plenum, in aworkspace, or in an area between the plenum and the workspace. Also, ifone plenum fan goes down, the other fans will simply speed up tomaintain the pressure differential and to thereby move the same amountof air. In other words, the fans across the plenum will generally sharethe increased load equally.

FIG. 2 c shows a perspective view of a modular computing environment 239for housing a data center or part of a data center. The modularcomputing environment 239 may be housed, in whole or in part, incontainer 208, which may be a standard shipping container, or may takeother appropriate forms. This figure better shows features in container208 to permit human occupancy, which may be of assistance in servicingthe computing environment in the container 208. Such human occupancy mayrequire additional features that satisfy both physical human occupancyrequirements and any legal requirements that may exist (e.g., municipalbuilding or occupancy requirements).

For example, the modular computing environment 239 provides a mechanismfor human ingress into and egress out of the interior of the enclosure(e.g., by doors 258 a, b). Lights 240 may be provided in the interior,as may be a source of fresh air and fire detection and suppressionsystems 242. The interior may be further designed within certaintemperature, humidity, and noise parameters, and a clearance may beprovided to allow human operators to move about the interior to maintainor service various components, such as shown in FIGS. 2A and 2B.

The modular computing environment 239 may further be designed to accountfor safety of human operators in the enclosure. For example, powersources may be covered and insulated to minimize risk of electricalshorting or shocks to human operators; fans may be enclosed withinprotective cages to contain fan blades that may cause injury when cominginto contact with, for example, a human finger or other appendage; andthe interior may be equipped with emergency lighting and exit signs 250.A removable, sectioned walkway (e.g., a metal grate (not shown)) may beprovided to allow human operators to walk along the length of themodular computing environment 239 and to access components below thewalkway (e.g., heat exchangers/coils or fans, in some implementations).

The systems may require, in certain implementations, features that meetphysical requirements of human occupancy (e.g., systems that regulatetemperature, noise, light, amount of fresh air, etc.) Moreover,depending on its location, the modular computing environment 239 mayfall under the jurisdiction of a local municipality as an inhabitablecommercial or industrial structure, and various systems may be requiredto meet local building or occupancy codes (i.e., legal requirements ofhuman occupancy).

As shown, the systems include lighting 240, fire/smoke detectors 242,and a fire suppression system 246. In some implementations, the firesuppression system 246 may include an inert chemical agent or water fogfire suppressant. The chemical fire suppressant may be stored in anon-board tank 244, or the water fog fire suppressant may be provided bya source external to the modular computing environment 239. A controller248 may cause the chemical fire suppressant to be released based oninput from the fire/smoke detectors 242. As shown, a number of firesuppressant heads are shown along the top of the modular computingenvironment 239; in some implementations, more or fewer heads mayprovided. For example, a fog-based fire suppression 246 system may onlyinclude a small number of heads. In some implementations, the firesuppression system 246 may be arranged in a different manner; forexample, each rack may include its own fire suppression subsystem.

Human occupancy regulations may require exit lights 250 and certainingress and egress parameters (e.g., number, placement and size of doors258 a, 258 b). Afresh air circulation system 252, 254 may also berequired to supply the interior of the modular computing environment 239with fresh air. Also, drains 256 may be provided to dispose of waterthat may enter the container or may leak, such as from cooling orsprinkler piping.

FIG. 3 a shows a plan view of a data center 300, and FIG. 3 b shows asectional view of the data center 300 from FIG. 3 a. Data center 300 issimilar to data center 200 from FIGS. 2 a-2 b. However, data center 300is shown located in a larger space, such as in a fixed building. Becauseof the additional space in this layout, racks of trays, such as tray320, are mounted back-to-back on common warm-air plenums 310, 312. Airfrom plenums 310, 312 is routed downward below a false floor and drivenback into workspace 308 by fans, such as fan 322, and through coils,such as cooling coil 324.

Again, an open platform 326 is provided on which workers in workspace308 may stand when monitoring or maintaining computers in the racks,such as rack servers. The rack servers may run from the floor up to ornear ceiling 304, which may be a drop tile ceiling, for example, at aheight of approximately 8 feet. The ceilings may also be much higher,such as if the plenums 310, 312 are capped.

In these figures, the racks are formed in bays having four racks foreach bay, and a per-bay length of approximately 6 feet. Other rackarrangements may also be employed so as to fit the needs and dimensionsof a particular data center implementation.

The pictured implementation and similar implementations may provide forscalability of data center 300. For example, additional rows of computerracks may be added in parallel, either to provide for a larger datacenter or to expand an existing data center. Relatively simplecirculation units (such as those just described) may be installed underthe racks and may require only water piping connections and electricalconnections for operating the fans. Thus, those components also permitfor simplified maintenance and installation of components for thesystem. For example, standardized components such as fan coil units maybe manufactured and assembled at a central location and then shipped toa data center worksite for installation. In addition, worn or brokencomponents may be removed and switched with newer components. Moreover,use of common components permits the operation of a data center withfewer replacement components on hand and also permits use of generalstock components available from many manufacturers and distributors.

Pressure sensors 309, 311 may also be provided to control the operationof fans 322. As shown, the pressure sensors 309, 311 are shown mounteddirectly in an endwall of plenums 310, 312, so as to provide adifferential pressure between a warm air plenum 310, 312 and a workspace308, but may be located in other areas, and may be provided withextension tubing so that sensing taps may be placed in any convenientlocation. The sensors 309, 311 may be used to control the system 300 inways like those discussed above for sensors 209, 211 in FIGS. 2A and 2B.For example, fans like fan 322 may be controlled to maintain a setpressure difference, such as zero, between the plenums 310, 312 and theworkspace 308.

FIG. 4 a shows a plan view of a data center 400, and FIG. 4 b shows asectional view of the data center 400 from FIG. 4 a. The data center 400is similar to data center 300 shown in FIGS. 3 a-3 b, but with thelocations of the fan-coil units changed. Specifically data center 400 isconfigured to be located in a fixed building and to be expandable inmanners similar to data center 300. In addition, data center 400, likedatacenter 300, includes pairs of racks having trays such as tray 420(which may include, e.g., a computer motherboard and associatedcomponents), mounted back-to-back, separated by intervening workspacessuch as workspace 408.

However, in data center 400, fan 422 and coil 424 are mounted at the topof warm air plenum 412, as are other fans and coils. In thisarrangement, air may be expelled from the fans across the ceiling andneed not turn as many corners as in the implementation of FIGS. 3 a-3 b.In other implementations, fan 422 may push air through the coil 424.Likewise, coil 424 may be located away from the racks so as to reducethe risk that water in the coil 424 will leak onto the racks. Inaddition, along the length of the racks, the fans and coils may point inalternating directions, so that some fans blow into the workspace to theright of a rack and some blow into the workspace to the left of therack.

Pressure sensors 409, 411 may also be provided to control the operationof fans 422. In the figures, the pressure sensors 409, 411 are shownmounted within plenums 410, 412 to sense the pressures in the plenums410, 412, and are provided with tubed extensions that sense pressure inthe workspace 408. The sensors 409, 411 may be used to control thesystem 300 in ways like those discussed above for sensors 209, 211 inFIGS. 2A and 2B.

FIG. 5 a shows a plan view of a data center 500, and FIG. 5 b shows asectional view of the data center 500 from FIG. 5 a. In thisimplementation, individual fan-coil units near each rack have beenreplaced with a single central unit in the middle of data center 500.The air that has warmed by passing through the racks may be drawn upwardthrough passages in the form of chimney 504, into an attic space 506.The chimney 504 may be in the form of a number of passages, such asround or rectangular ducts, or may be in the form of a passage that runsthe entire length of a row of trays, or may take another appropriateform. Advantageously, the attic space may be naturally warm from theoutdoor environment and from radiated heat transfer from the sun, sothat little to no heat will be transmitted through the roof into theattic. In appropriate circumstances, the warmed air entering the attic506 may be at a higher temperature than the outdoor temperature, andheat transfer may occur out of the attic, rather than in.

Warm air is drawn from attic space 506 by supply fan 502, which may belocated above an area near the center of data center 500. Locating fan502 in attic space 506 may reduce the noise level transmitted from fan502 into workspace 508. Other sound insulation techniques may also beused such as by insulating the ceiling of workspace 508. Such insulationmay also provide thermal insulation that prevents heat from passingdownward through the ceiling.

Fan 502 may connect to a plenum 514, such as a plenum that takes theplace of a pair of back-to-back bays. The plenum 514 may be formed froman enclosure that seals the plenum 514 from adjacent racks. The sides ofthe enclosure may be covered with cooling coils such as coil 510.Cooling water may be passed through cooling coil 510 under conditionslike those discussed above. Cooling coils 510 may be sized and selectedto have relatively shallow fins, so as to present a minimal pressuredrop. Fan 502 may alternatively be located, for example, in the areataken up by plenum 514 in the figure.

Mechanisms for obtaining outdoor air into the attic 506 or another areamay also be provided. For example, motor-controlled louvers to theoutdoors may be provided and may be caused to take in outside air whenatmospheric conditions are favorable (e.g., low temperature and lowhumidity). Air filtering may be performed on incoming fresh air. Suchfresh or outside air may be blended with heated, re-circulating air fromthe space also. The amount of such blending may be controlledelectronically to produce desired temperature or other values in thespace. In addition, separate air-conditioning units may also be providedto provide supplemental or spot cooling, and to remove any built-uplatent heat. When operating in such an air-side economizing mode,exhaust fans may also be provided to remove, for example, heated air,and may be controlled via differential pressure sensors to coordinatetheir operation with other components in the system. In addition,appropriate filtration and humidity control may be provided for airentering a system from the outdoors.

Heat transfer to cooling coil 510 may be improved where heat risethrough the trays is high, and air-flow volumes are therefore relativelylow. As a result, the volume of air moving through cooling coils 510will also be relatively low, so that additional heat may be transferredfrom the warm air into the cooling water in cooling coil 510. Protectivepanel 512, such as a louver or a protective wire mesh screen, may beprovided in front of cooling coil 510 to prevent workers fromaccidentally bending fins in the cooling coil. In addition, redirectingvanes (not shown) may be provided in plenum 514 to direct air laterallythrough cooling coils 510, 512.

Pressure sensor 505 may also be provided to control the operation of fan502. In the figures, the pressure sensor 505 is shown mounted withinplenum 506 to sense the pressure in the plenum 506, and is provided witha tubed extensions that passes into the workspace to sense pressure inthe workspace 408. The sensor 505 may be used to control the system 500in ways like those discussed above for sensors 209, 211 in FIGS. 2A and2B.

FIG. 5 c shows a sectional view of another implementation of a datacenter. In this implementation, a number of fan coil units 534 areprovided in an attic space 536. The units may also be located, forexample in a below floor, or basement, space. As in FIG. 5B, variousracks of electronic equipment, such as rack 530, deliver warm air intothe attic 536 through various chimneys 540. The chimneys 540 and fancoil units 534 may be positioned so as to avoid interfering with eachother and to avoid unwanted pressure changes in the attic 536 orelsewhere. (In addition, basements may be used in a similar manner toattic 536 described here.)

The fan coil units 534 may take any appropriate form, such as commercialfan coil units containing a standard cooling coil and a centrifugal orother form of fan. The fan coil units 534 may each be connected tosupply air ductwork that empties into workspace 538. The ductwork may,for example, terminate in diffusers of various forms. The supplyductwork from multiple units 534 may be interconnected to permit forswitching or for redundancy if one unit goes down.

One should understand that the units 534 may be laid out intwo-dimensions in the attic, so that, for example, the leftmost unit inthe figure is not blocking chimney 540. Instead, it may be in front orbehind of chimney 540 in the figure. A catwalk 532 may also be providedin attic 536 so as to provide more ready access to units 534.Alternatively, or in addition, provisions may be made to service units534 from workspace 538.

Pressure sensors 533 a-c may also be provided to control the operationof fans coil units 534. In the figures, the pressure sensors 533 a-c areshown mounted within a fan coil cabinet, in a plenum or an exhaust tubefor a plenum, and in the open attic plenum 536 to sense the pressures inthe plenums 536 and are provided with tubed extensions that sensepressure in the workspace 538. The sensors 533 a-c may be used tocontrol the system 300 in ways like those discussed above for sensors209, 211 in FIGS. 2A and 2B.

FIG. 6 is a flowchart showing actions for an exemplary operation ofcooling components in a data center. In general, the depicted process600 involves steps for adding computers to a rack-mount computer system,and to control the flow of air over those computers so as to maintaintemperatures and temperature changes to permit energy-efficientoperation of the system.

At box 602, an operator connects one or more rack-mount servers to awarm air plenum. Such actions may occur by sliding or rolling apre-loaded rack into the plenum, such as by placing the rack in front ofan opening in one wall of the plenum. Alternatively, a rack that isempty or partially full may be moved into location, and additional traysmay be added to the rack. Where appropriate, additional steps may betaken to seal around the edges of the racks or the trays in a rack. Forexample, blanking panels may be provided where trays are missing from arack so as to prevent short-circuiting of air into the warm air plenumat those locations.

At box 604, the servers are started, as is a fan-coil unit or multiplefan-coil units (which may include packaged units or combinations of fansand cooling coils that are not in a pre-built unit). Starting theservers may entail powering up the various components that support themicroprocessors on the servers, including chipsets and hard drives. Eachserver may be started at the server itself, or may be powered upremotely such as from a central control unit. Operation of the fan-coilunit or units may occur through an HVAC control system that may beconfigured to sense various parameters related to the controlledenvironment of the data center and to regulate the operation ofcomponents, such as fans and pumps that operate to regulate temperaturein a space or spaces. Where multiple fan coil units are employed, theymay be located in a mezzanine or attic space, which may itself serve asa warm-air plenum, and may provide cooled air to a workspace.

At box 606, a new rack is added to a warm air plenum. For example, ablanking panel may initially be located over an open space in theplenum, the panel may be removed, and a rack loaded with trays may beslid into place and sealed against the plenum. That rack may then beconnected appropriately, such as by providing a power connection from acentral source of power, and a networking connection (or multipleconnections). Upon making the necessary connections to the trays, suchas server trays, and the rack, exhaust fans for pulling air over eachtray and into the warm air plenum may also be started at this point, ormay be provided power but may delay their start until a particulartemperature for a tray is reached. The start-up of each tray may becontrolled remotely or locally, as may the control of other componentsin the system.

At box 610, the speeds of exhaust fans serving the various trays may becontrolled to maintain a particular exit temperature at each tray. Thespeeds may be selected to slow the airflow to a rate lower than wouldtypically be used to cool electronic equipment. The temperature may beselected to produce a particular temperature rise across the computers,where the input temperature is known. For example, as noted above, ifthe temperature of a workspace is known to be approximately 25 degreesCelsius, then a temperature rise of 20 degrees Celsius may be maintainedby holding the exhaust temperature to 45 degrees Celsius.

At box 612, a fan-coil unit may likewise be controlled to maintain atemperature for the workspace. As one example, the fan in a fan-coilarrangement may be modulated using sensors that measure a pressuredifferential between a warm-air plenum and a workspace, such as tomaintain a 0.1 or 0.002 inch pressure difference between the spaces. Thepumping rate of cooling water may then be modulated to maintain a setworkspace temperature. Also, multiple controls may be aggregated andcontrolled from a central building management system.

Where pumping cannot meet the load, additional cooling may be provided,such as from a chiller or similar cooling equipment. However, suchsupplemental cooling will generally not be required, and may only occuron particularly hot or humid days (in which a cooling tower alone cannotsufficiently cool the cooling water), or when load is particularly high.Other free cooling sources (i.e., for cooling with no or almost nochiller operation) or air-side economizer sources other than coolingtowers may also be used, such as deep lake and ocean cooling.

FIG. 7 shows plan views of two exemplary trays for use in a rack-mountcomputer system. Tray 700 a hosts a number of storage devices, such asfixed disk drives, while tray 700 b includes a computer motherboardholding various computing components. Both trays may be centered aroundcircuit boards, or motherboards, that hold the various components, andon which may be formed conductive traces for electrically connecting thecomponents. Other components may also be provided on the trays, such assupporting chip sets and various forms of connectors, such as SCSI orATA connectors that may tie tray 700 a to tray 700 b.

Referring now to each tray 700 a, 700 b individually, tray 700 acontains memory 724 near its front edge, as memory generates relativelyless heat than do microprocessors 721 a, 721 b located downstream. Thememory 724 may be located in line with the airflow so as to permit moreflow over tray 700 a. In addition, network connection 722 may also bemounted at the front edge of tray 700 a so as to permit ready connectionof the tray 700 a to other portions of a rack-mount system. Anelectrical connector (not shown) may also be provided on the front edgeof the tray 700 a.

Microprocessors 721 a, 721 b may be located below impingement fans 720a, 720 b, in a manner similar to that discussed above. Supporting chipsets and other components may be located next to or near theirrespective microprocessor.

Walls 726 a, 726 c channel airflow over tray 700 a, and through powersupply 714 and fan 716. As arranged, components that generate more heatare placed closer to the fan, as are components (like the power supply714) that are less sensitive to high temperatures.

Tray 700 b holds a number of hard drives 718 a-718 c and is dedicated tostorage. Because the hard drives 718 a-718 c generate relatively littleheat, tray 700 b is not provided with a fan. Although a power supply isnot shown, so that tray 700 b may share power from another source, tray700 b may also be provided with a power supply on its back edge. Thepower supply may be provided with a fan or may be allowed to run hot.

Cable connections (not shown) may be provided between the hard drives718 a-718 c on tray 700 b and the components on tray 700 a. For example,standard ATA, SATA, or SCSI cable connections, or other appropriatecable connections, including high-speed custom cable connections may beused to permit high data rate transfers from hard drives 718 a-718 c.Serial cable connections may provide for better airflow than mayribbon-type parallel cable connections.

Trays 700 a, 700 b may take on a reduced-size form factor. For example,each of trays 700 a, 700 b may be approximately 19 inches in length andapproximately 6 inches or 5 inches wide. Multiple trays may be part of asingle motherboard or may be connected side-by-side to fit in a widerslot in a rack. In such a situation, each tray may be self-contained,with its own power supply and fan (as shown), or the trays may sharecertain components. Other similar sizes may also be employed so as tofit in existing rack systems.

A number of embodiments have been described. Nevertheless, it will beunderstood that various modifications may be made without departing fromthe spirit and scope of the disclosures in this document. For example,additional components may be added to those shown above, or componentsmay be removed or rearranged. Also particular values for temperaturesand other such values may be varied. Moreover, steps in processes may berearranged, added, or removed as appropriate. Accordingly, otherembodiments are within the scope of the following claims.

1-48. (canceled)
 49. A system for cooling a computer data center,comprising: a data center building that defines a workspace and an atticspace; a ceiling of the workspace that physically separates theworkspace form the attic space; a plurality of computer racks housingcomputer systems and arranged in the workspace to receive supply aircirculated from the workspace; a warm air capture plenum fluidlyconnected to the attic space and arranged to receive warmed air from thecomputer racks and; one or more outdoor air intakes arranged to receiveoutside air into the data center building; and one or more exhaust fansarranged to remove warmed air from the data center building.
 50. Thesystem of claim 49, further comprising one or more differential pressuresensors to control operation of one or more air circulation fans thatcirculate the supply air, wherein the differential pressure sensors arepositioned to sense relative pressure between the attic space and theworkspace.
 51. The system of claim 49, wherein the supply air circulatedover the computers is received from walkways in the workspace inside thedata center building.
 52. The system of claim 49, wherein the workspacedefines a cool air plenum that is thermally isolated from the warm aircapture plenum.
 53. The system of claim 49, wherein the warm air captureplenum is located between back sides of adjacent, parallel rows ofcomputer racks.
 54. The system of claim 49, further comprisingmotor-controlled air intake mechanisms arranged to cause outside air tobe taken into the data center building based on outdoor atmosphericconditions.
 55. The system of claim 49, further comprising one or moreair blenders arranged to blend outdoor air with the heated air exhaustedfrom the data center building, wherein the blended air is circulated tothe workspace and through plurality of computer racks.
 56. The system ofclaim 49, wherein a roof of the data center building is sloped along twoplanes symmetrically about the midline of the data center building. 57.The system of claim 49, wherein the warm air capture plenum is in fluidcommunication with the attic space via one or more chimneys locatedabove the warm air capture plenum.
 58. The system of claim 49, whereinthe warm air capture plenum is one of a plurality of warm air captureplenums arranged along an entire length of a row of computer racks. 59.The system of claim 49, further comprising a fan unit located in theattic space to circulate the heated air form the warm air captureplenums.
 60. The system of claim 49, further comprising a controller tooperate the system in an airside economizer mode.
 61. The system ofclaim 49, further comprising a plurality of trays in each of thecomputer racks, the plurality of trays each open to the workspace andadjacent to the warm air capture plenum and having a computermotherboard.
 62. The system of claim 61, wherein particular ones of theplurality of computer trays are fanless computer trays.
 63. A method ofcooling a computer data center, comprising: receiving outdoor air into adata center building; circulating cool air over computers mounted in aplurality of computer racks and into a warm air capture plenum adjacentthe computer racks to warm the cool air; circulating the warm air fromthe warm air capture plenum into an attic space above the computerracks; and exhausting at least some of the warm air to outside of thedata center building.
 64. The method of claim 63, further comprisingusing one or more differential pressure sensors to control operation ofone or more air circulation fans that circulate the air.
 65. The methodof claim 64, wherein the differential pressure sensors sense relativepressure between the attic space and the workspace.
 66. The method ofclaim 63, further comprising circulating the supply air from walkways ina workspace of the data center building over the computers mounted inthe plurality of computer racks.
 67. The method of claim 66, wherein theworkspace defines a cool air plenum that is thermally isolated from thewarm air capture plenum.
 68. The method of claim 63, wherein the warmair capture plenum is located between back sides of adjacent, parallelrows of computer racks.
 69. The method of claim 63, further comprisingcirculating the warm air from the warm air capture plenum to the atticspace through a chimney located above the warm air capture plenum andfluidly connecting the warm air plenum with the attic space.
 70. Themethod of claim 69, further comprising circulating all of the warm airreceived into the warm air capture plenum from the computer racksthrough the chimney and to the attic space.
 71. The method of claim 63,wherein a temperature of the warm air entering the attic space from thewarm air capture plenum exceeds a temperature of the outdoor air. 72.The method of claim 63, further comprising circulating the warm air fromthe warm air capture plenum to the attic space via a fan unit located inthe attic space.
 73. The method of claim 63, further comprising: settinga control system of the data center building to an airside economizermode; and in the airside economizer mode, circulating the outside airover computers mounted in the plurality of computer racks and into thewarm air capture plenum adjacent the computer racks.
 74. The method ofclaim 73, wherein only outside air is circulated over computers mountedin the plurality of computer racks in the airside economizer mode.
 75. Asystem for cooling a computer data center, comprising: a data centerbuilding that defines a workspace and an attic space; a ceiling of theworkspace that physically separates the workspace form the attic space;a plurality of computer racks housing computer systems and arranged inthe workspace to receive supply air from the workspace; means forcirculating cool air from the workspace across computers in the computerracks to heat the cool air, and into a warm air capture plenum that isfluidly connected to the attic space; one or more outdoor air intakesarranged to take outside air into the data center building; and one ormore exhaust fans arranged to remove the heated air from the data centerbuilding.
 76. The system of claim 75, wherein the cool air circulatedacross the computers is received from walkways in a workspace inside thedata center building.
 77. The system of claim 75, further comprising achimney arranged along a length of one or more rows of the computerracks that fluidly connects the warm air capture plenum to the atticspace.
 78. The system of claim 75, further comprising a fan located inthe attic space to circulate the heated air from the warm air captureplenum.